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Continuation-in-Part Application 
A Video Camera Utilizing Sequential Diversity Imaging For Image Clarification 

Background of the Invention 

The present invention is a Continuation-in-Part of Application No. 09/905,693 entitled, 
"Sequential Diversity Imaging," filed on November 14, 2000, and it relates to imaging 
through a random optical medium, such as the turbulent atmosphere or the slowly 
changing characteristics of a video camera. 

Wavefront sensing is employed to determine the distorting wavefront caused by the 
turbulent atmosphere and one or more the following methods are generally employed 
to provide an estimate of the distorting wavefront. . 

1 . Dithering. This method of wavefront sensing continuously changes the adaptive 
optics and monitors the image quality of the observed image. US Patent 3,979,585 
entitled "Adaptive imaging telescope with camera-computer transform image quality 
sensing and electro-optic phase shifting " and the publication by M. A. Voronstov and 
V. P. Sivokon, entitled "Stochastic parallel-radiant-descent technique for high-resolution 
wave-front phase-distortion correction," J. Opt.Soc.Am.,A,1 5,2745 (1988) are examples 
of this approach. They are sequential techniques, in the sense that a physical property 
of the adaptive optics is changed, and if the next (time-sequential) image is sharper the 
change is increased, otherwise the change is reversed. This is a physical search for 
"best focus" and is how the human eye and most cameras work. 



2. Shearing Interferometer. This method of wavefront sensing uses a reference 
beam to create an interference pattern, from which the unknown wavefront is estimated. 
It requires a laser-based interferometer. US patent 3,923,400 entitled "Real-time 
wavefront correction system " describes this approach. 

3. Shack-Hartmann Sensor. This device employs an array of lenses to focus 
multiple, small images, each seen through a different section of the aperture, onto a 
detector. Shifts in the small images are caused by local tilts in the waveform, which 
allows the wavefront to be reconstructed. US Patents 4,141 ,652 entitled "Sensor 
system for detecting wavefront distortion in a return beam of light" and 5,350,91 1 
entitled "Wavefront error estimation derived from observation of arbitrary unknown 
extended scenes" disclose the use a Shack-Hartmann sensor. 

4. Curvature Sensing. This method is described in the publication by F. Roddier et 
al. entitled "A simple low-order adaptive optics system for near-infrared applications," 
Publications of the Astronomical Society of the Pacific, 1 03, 1 31 (1 991 ), whereby two or 
more images are measured along the path of the optical system. The local curvature of 
the propagating wave is determined and it is propagated, by computer calculation, back 
to the aperture to form the wavefront estimate. 

5. Phase diversity. This method employs diverse images, measured 
simultaneously whereby the diversity is a quadratic phase shift, which can be introduced 
by defocusing the optical system. Additional equipment in needed to record the out-of- 
focus image. US Patents 4,309,602 entitled "Wavefront sensing by phase retrieval", 
5,384,455 entitled "Measurement-diverse speckle imaging", 5,610,707 entitled 
"Wavefront sensor for a staring imager", 6,107,617 entitled "Liquid crystal active optics 
correction for large space based optical systems ", and the publication by R. Paxman, 
et al. entitled "Optical misalignment sensing and image reconstruction using phase 
diversity," J. Opt. Soc. Am., A, 5, 914 (1988), each use phase diversity to estimate the 
wavefront. It is noted that the phase diversity approach to wavefront sensing was used 
to determine the aberration in the Hubble telescope project. 



Note that all of the patents mentioned in the section on phase diversity use an in-focus 
image, an out-of-focus image, and a fixed diversity to perform wavefront sensing, as 
opposed to the current invention which uses sequential, in-focus images and sequential 
diversities which are the sequential changes in the adaptive optic device. The 
invention does not need additional equipment , such as an out-of-focus image, to 
control an adaptive optic device in the camera system. 

One purpose of the instant invention is to provide a system for determining aberrations 
within a video camera or the optical medium and to eliminate the aberrations using no 
additional optics or sensors . 

Summary of the Invention 

The invention describes a video camera which produces digital clarified images of an 
object which is distorted by time-varying aberrations in the optical medium. It uses an 
adaptive optics device to eliminate the aberrations and a sequential diversity processor 
to produce the control signals for the adaptive optic device. The sequential diversity 
processor uses sequential, in-focus frames of the camera, each with a known diversity 
such as phase, wavelength, or spatial shift, to deduce both the unknown object and the 
aberrations in the optical medium. Sequential frames are the diverse images and the 
changes in the adaptive optics, per se, are used as the diversities. Unlike other diversity 
Imaging methods no other auxiliary equipment or signal, such as an intentionally 
defocused image, is needed to control the adaptive optic device. 



Brief Description of the Drawings 

Figure 1 is a diagrammatic representation of the imaging system, which employs an 
adaptive optic and a sequential diversity processor within a video camera, in 
accordance with the invention; 



Figure 2 is a flow chart representation of the imaging system and the sequential 
diversity algorithm used within the sequential diversity processor of Figure 1 ; and 
Figure 3 depicts a computer simulation of 6 frames of a video sequence achieved within 
the video camera of Figure 1 . 

Description of the Preferred Embodiment 

The sequential diversity imaging system 12 within the video camera 13 depicted in 
phantom in Figurel, receives an optical signal 1 which may have aberrations introduced 
by a randomly changing optical medium. The optical signal passes through the camera 
aperture 2 and is imaged by a lens 3. Between lens 3 and detector 5 is an Adaptive 
Optic 4, hereinafter, "AO". This portion of the invention is called the optical system, as 
described in the publication entitled "Phase retrieval and diversity in adaptive optics," 
R.A. Gonsalves Opt. Eng., 21, 829 (1982) and as further described in US Patent 
4,309,602. The optical system produces a clarified in-focus optical image of the 
distorted object In the focal plane of the lens 3. The AO 4 modifies the optical 
properties of the optical energy it receives from the lens so as to cancel aberrations in 
the object being viewed. It also receives digital signals from the sequential diversity 
processor 7, signals which control the AO 4. 

The output of detector 5 is a video sequence of digital images, as indicated at 6. This 
sequence of digital in-focus images is the input to a Sequential Diversity Processor 7 
which produces digital control signals 8 to control the configuration of the AO, so as to 
cancel the aberrations introduced by the random optical medium. The AO could 
comprise a high-resolution device to allow a wide range of correction mechanisms, such 
as Zernike polynomial fitting of a complicated wavefront. 

Figure 2 depicts the image flow through the lens 3, AO 4, detector 5 and the 
processing of data within the Diversity Algorithm 10 and the Predicted Algorithm 1 1 . 
The k th frame of the video output 6, l(k), is an input to the Diversity Algorithm 1 0. Other 
inputs to the Diversity Algorithm 10 are successively delayed versions of l(k), namely 
l(k-1), l(k-2), which are delayed and stored in a digital buffer Delays 9A . The k th 



output of the Diversity Algorithm, D(k), and delayed versions from Delays 9B , are 
used as inputs to the Diversity Algorithm 10. D(k) is an input to the Predicted 
Algorithm 1 1 which predicts the change in the random optical medium for the next 
frame of the video sequence and controls the AO 4 with a control signal T(k) as 
indicated at 8. The other inputs to the Predicted Algorithm 1 1 are delayed versions T(k- 
1), T(k-2) ,— , from Delays 9C, of the control signal T(k). 

To further describe the flow diagram of Figure 2, the following notations assume that the 
media aberration is due to an unknown wavefront distortion, which is typical of 
atmospheric distortion: 

W(k) = Unknown distorting wavefront at time k. 

T(k) = Phase on the AO at time k. 

C(k) = Residual phase to be estimated 

= W(k)+T(k). (1) 

l(k) = Measured image at time k. 

D(k) = Diversity phase. 
With l(k-1 ) as the first image and l(k) as the diversity image, the diversity phase D(k) is 
the change in the AO phase from time k-1 to time k, such that: 

D(k)= T(k)-T(k-1). (2) 

Let Q(k) represent the estimate of C(k), as in equation (1 ), 

Q(k) = W1(k) + T(k) , (3) 

where W1 (k) is an estimate of W(k), the unknown phase at time k. At time k + 1 the 
AO should be set to the negative of the unknown distorting wavefront W(k+1) at time 
k+1 . An estimate of the distorting wavefront at time k, W1 (k), provides a good estimate 
of W(k+1 ) when AO updates are within the time constant of the changing optical 
medium. Therefore: 

T(k+1)= -W1(k), (4) 
which tends to cancel the wavefront distortion at k+1 . 
Solving (3) for W1(k) and substituting into equation (4), results in 



T(k+1)= - Q(k) +T(k) ,which implies 
T(k) =-Q(k-1) +T(k-1) . 



(5) 



To find the diversity phase D(k) insert equation (5) into equation (2) whereby 

D(k) = ( - Q(k-1 )+ T(k-1 ))- T(k-1 )= - Q(k-1 ) . 
The new diversity D(k) is now defined as 

D(k) = -Q(k-1), (6) 
which provides the diversity at time k. 

Inserting equation (6) into equation (5) results in the new AO control signal: 

T(k) = D(k) + T(k-1), (7) 
which is a simple restatement of equation (2). 

Equations (6) and (7) result in the sequential diversity imaging function in accordance 
with the invention. 

In summary, the sequential diversity algorithm 10 and predicted algorithm 11 of Figure 
2 contained within the sequential diversity processor 7 of Figure 1 employ the diversity 
D(k-1) and the diverse images, l(k) and l(k-1), to calculate an estimate Q(k-1) of the 
residual phase C(k-1 ). ,- The new diversity, D(k), is set to the negative of Q(k-1 ), 
according to equation (6), and it is added to the previous AO setting to form the new 
AO setting, T(k), according to equation (7). 

Figure 3 depicts a computer simulation of frames 1-6 as achieved within the sequential 
diversity processor 7 within the video camera 13 of Figure 1 . The object 17 comprises 
four lines of text viewed through a turbulent atmosphere 1 via the video camera lens 3. 
The original unprocessed images 15 of the text are shown in lower regions of each 
frame and the processed images 16 are shown in the upper regions thereof. 
Comparing the original images 15 to the processed images 16 within each frame 
depicts how the processed images are clear and readable within 6 frames. The Strehl 



ratio, the ratio of the peak value of the unprocessed Point Spread Function, "PSF" to 
the peak value of the processed PSF as described in the aforementioned publication 
"Phase retrieval and diversity in adaptive optics," wherein higher Strehl ratio defines 
higher image quality is now employed to determine image quality for the original 
unprocessed images 15 and the processed images 16. For the 6 original unprocessed 
images depicted in Fig. 3, the average Strehl ratio is 0.09 whereas the average Strehl 
ratio for the processed images is 0.48, which results in an improvement of about a 
factor of 5. 

Accordingly, the invention teaches sequential diversity imaging within a video camera 
for determining the aberrations caused by the optical medium and for controlling an 
adaptive optic device in the optical system to eliminate the aberrations. The adaptive 
optic device clarifies the image as it appears on the detector in the lens' focal plane and 
the detector produces a sequence of digital clarified in-focus images. These solely in- 
focus images are the input to the sequential diversity processor which estimates the 
residual phase emerging from the AO, the diversity to be used in the next cycle of the 
diversity algorithm, and the control signal for the AO device. No out-of focus images are 
needed nor is an additional detector needed to measure a defocus image, as is the 
case with conventional phase diversity systems identified in the above section, entitled 
Background of the Invention. 



